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Watennaik CTibedding and detection of a motion image signal. 



FIELD OF TEE INVENTION 

The invdition relates to a method and arrangement for embedding a 
watermark in motion image signals such as movies being projected in cinemas. The invention 
also relates to a metibiod and arrangement for detecting a watermajdc being embedded in such 
motion image signals. 

BACKGROUND OF THE INVENTION 

Watermark embedding is an important aspect of copy protection strategies. 
Although most copy protection schemes deal with protection of electronically distributed 
contents (broadcasts, storage media), copy protection is also desired for movira being shown 
in theaters. Nowadays, illegal copying of cmema material by means of a handheld video 
camera is already common practice. Although the quality is usually low, the economical 
impact of illegal VHS tap«, CD-Video's and DVD's can be enormous. For this reason, 
cmema owners are obliged to prevent the presence of video cameras on their prranises. Not 
following this rule may be sanctioned with aban on the foture availability of content In view 
hereof it is envisioned to provide that a watermark will be added during show time. The 
watermark is to identify the cinema, the presentation time, operator and such. 

Robustness to geometric distortions is a key requirement for such watermark 
embedding schemes. A handheld camera will not only seriously degrade the video by 
filtering (the optical path from the screen to the camera, transfer to tape eto.) but also 
seriously geometrically distort the video (shifting, scaling, rotation, shearing, changes in 
perspective, eto.). In addition tiiese geometrical distortions can change fiom fiame to fiame. 

A prior art method of embedding a walermaik in cmema movies, which meets 
the robustness requirements, is disclosed in Jaap Haitsma and Ton Kalken A Wat ermarking 
Scheme for Digital Cmema; Proceeduigs ICIP, Vol.2, 2001, pp. 487-489. The robustness to 
geometric distortions is achieved by exploitmg only the tenoporal axis to embed the 
watermark. The watermark is a periodic pseudo-random sequence of watermaric san^les 
having two distinct values, e.g. and One watermark sample is enibedded in each 



PHNL030093EPP 



2 20.01^003 
image. The value ' r is embedded in an image by inCTeasing a global properQr (e.g. Ihe mean 
luminance) of Ihe image, the value '-1' is embedded by decr«ising said global property. 

The pnor art watenxiaj^ embedding melbod actually embeds £i<^ 
embedding tiie same watermark sample in a number of consecutive images, the fUcker is 
5 made inqietceptible (the human eye is less sensitive to low-fiequency flicker). 

Flicker of the recorded movie is also caused by a) the typical migma^ Tii 
betvreen the cinema projector's fiame rate (24 fiames per second) and tte camcorder's ftame 
rate (25 ^ fi>r PAL, 29.97 fys for NTSC), and b) Ae difGatence between the two display 
scan formats (progressive vs. intedace). This kind of flicker is so annoying tiiat de-flickering 
10 tools have been made widely available to ihe public. For exaniple, a de-flicker plug-in for the 
video cf^turing and processing plication "Virtualdub" has been found on the Internet. 

A problem of the prior art watermark embedding scheme is that de-flicker 
tools also remove the embedded watermark. 

15 OBJECT AND SUMMARY OF TEffi INVENTION 

It is an abject of tiie inveotion to forther improve the prior art watermark 
embedding and detection method. It is a particular object of the invention to provide a 
watetmaik embedding and detection scheme, which is robust to de-flickering operations. 

To this end, the method of embedding a watemoKitk in a motion image signal 
20 according to the invention includes dividing eadh image into at least first and a second image 
area. One value of a watermark san^le is embedded in an image by increasing the global 
properly (e.g. the mean luminance) of its first area and decreasing the global property of its 
second area. The other value of the watermaric sanq)le is embedded in the opposite way, i.e. 
by decreasing the global property of tiie first image area and increasing the global property of 
25 the second image area. 

The invention exploits tiie insigiht that de-flickering tools remove flicker by 
adjusting the mean luminance of successive images to exhibit a low-pass character. The mean 
luminance is adjusted in practice by multiplying all pixels of an image by the same foctor. 
Because tiiis operation does not a£fect the (sign of the) modifications applied to the distinct 
30 image areas, tiiewateriiHirkinfonnation win be retained. In view hereof Ihe global property 
of an image area being modified to embed the watermark is the mean hnninance of said 
image area. 



PHNL030093EPP 



3 20.01,2003 
In a preferred embodiment of the method, the first and second image area are 
the iq»per and lower half of an image. In general, Iheie are more horizontal than vertical 
movCTients in a movie. The horizontal movements influence the mean luminance values less. 

5 BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 shows a schematic diagram of a watemiark enibedder in accordance 
with iSie invention. 

Fig. 2 diows a schematic diagram of a watermark detector in accordm 

the invention. 

10 Fig. 3 shows a schematic diagram of a correlation stage, which is an element 

of tiie watermark detector shown in Fig. 2. 

Fig. 4 shows gr^hs of the mean luminance values of an original image 
sequence and watermarked image sequence. 

15 DESCaUPTIONOFENmODIMElSrrS 

Fig. 1 shows a schematic diagram of a watermark enibedder in accordance 
witii tiie invention. The embedder receives a sequence of images or fiameshav^ 
lumiiiance F^^k) at spatial position n of fiame k. The embeds 
in the form of a pseudo-random sequence w(n) of length N, where w(n)€ [-1, 1]. An 
apptapnate value of N fixr this s^Ucation is N=1024. The arrangement comprises a dividing 
stage 10, which divides each image into a first (e.g. upper half) area and a second (e.g. lower 
halQ area. The luminance of said image areas are denoted Fi(njk:) and Fiin^, resfpectively. 

In the sin^lest embodiment of the watermark enibedder, the sequence w(n) is 
directiy ^>plied to embedding stages 11 and 12. In such an embodiment, ^bedding stage 11 
adds one ^iplied watermark satnple w(n) to every pixel of tiie fust image area, whereas 
embedding stage 12 subtracts the same watermark sample firom every pixel of the second 
image area. Clipping is performed where necessary. The mean luminances of the first and 
second image areas are thus oppositely modulated by the watermark. 

Other examples of global irciage properties that can be modulated by the 
watermark are pictm^e histograms (a list of relative fiequendes of luminance values in the 
picture), or features derived therefirom such as hig^ order moments (average of luminance 
values to apower k). The means luminance is a specific example of the latter (k=l). 

Since tiie Human Visual System (TSVS) is seositive to flicker in low spatial 
fiequendes, tins sinxple embodiment may suffer fiom artr&cts in especially non-moving flat 
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areas. These arti&cts are sigoificanfiy reduced by lowering the flicker frequency of the 
watermark. This is performed by a repetition stage 13, which repeats each watemiadc sanqple 
during a predetermined number K of consecutive images. The same watermaric sample is thus 
enibedded in K consecutive fiames. The watermark repeats itself every N=1024 £tames. The 
watermadc sample w(n) being embedded in frame k can mathematically denoted by 
w( [k/KjmodN) . For simplicity, this expression will hereinafter be abbreviated to w(k). 

The preferred embodiment of the embedder which is shown in Fig. 1 further 
ad^ts the embedding depth in dependence upon the injSLgd contents. To this end, the 
embedder conrpris^ multipliers 14 aad 15, which multiply the watermark sample w(k) with a 
local scaling factor Cp.i^yk) and CF^^,k), respectively. The local scaling factors are derived 
form the image contents by image analyzers 16 and 17, respectively. For example, Ihey are 
large in moving textured parts of an area and small in non-moving flat parts . The outputs of 
the embedding stages 11 and 12 can be formulated as: 

F^4fe,k) = Fi(n,k) + CF,i(n,k)w(k) 

Fw^(a,k) = F2fe,k)-CF,2(a,k)w(k) 

It will be appreciated, that both einbedding operations can be carried out in a time sequential 
manner by a single processing circuit under s^ropriate software control. 

The two image areas are subsequently combined by a combining stage 18 into 
a single watermarked image F^v(n,k). 

Fig. 2 shows a schematic diagram of a watermark detector in accordance with 
the invention. Although the original signal is available during detection, the detector does not 
utilize any knowledge about the original. The arrangement receives a watermacked sequence 
of images or fiames having a luminance Fw(n,k) at spatial position n of fiame k. The detector 
conquises a dividing stage 20, which divides each image into a first (e.g. upper halQ area and 
a second (e.g. lower half) area in a similar manner as dividing stage 10 (Fig. 1) of the 
embedder. The luminance of each image area is denoted Fw,i(S5k) and Fw^(n,k), respectively. 
For each image area, Ifae detector further includes a mean luminance confuting circuit 
21, 22, which compute the mean luminance values ^,iQO ^d fw^(k) (or other global 
property, if applicable) of the respective image areas in accordance witii: 

fw.i(k)=^EFw,i(S,k) 
n 

Inixractice, tiie mean luminance values of amovie exhibit a low-pass nature as 
function of the ftame number k (ie. as function of time). The detector estimates tiie msan 
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lunuBance values of the origmal (unwatermaik^) movie by low-pass jBltering (25, 27) Hie 
respective mean luminance values fw,iQ^ and fw^(k). Estimations of Ihe.mean luminance 
modifications as introduced by the embedder aie subsequently obtained by subtracting 
(26, 27) the low-pass filtered mean values from the unfiltered mean luminance values. The 
detector esthnales the embedded watermark sample by sia>tracting (23) bo 
followed by a sign operation (29). The ratibtnated watermark saniple being embedded in fisme 
k is denoted v(k). 

The arrangement thus generates a sequence of estimated watermark samples. 
In a correlation stage 3, the estimated sequence of watermark san^les is correlated with the 
watermark being looked for. The detector receives this watermark being looked for in the 
form of a pseudo-random sequence w(n) of length N, where w(n)e [-1, 1]. The detector 
comprises a repetition stage 24, which is identical to same repetition stage 13 of tiie 
embedder. The repetition stage repeats each watermark sample during K consecutive images. 
The watermark repeats itself every N»1024 fitimes. The watermark sanq)les bdng applied to 
the correlation stage 3 for fi:ame is denoted wQc). Again, w(k) is an abbreviation fiir the 
mathematically more correct expression w( [k/KjmodN) . 

It should be noted that the low-pass filter / siibtractor combinations 25,26 and 
27^, as well as the sign operation 29 are optional. 

Fig. 3 shows a schematic diagram of the correlation stage 3. The estimated 
watermark samples successive images are distributed to K buffers 31, 32, where, as 
described above, K is the number of consecutive images in which the same watermark 
sample is embedded. Each buffer stores N estimated watermark samples (or N computed 
mean Imninance values, or N estimated mean luminance modification values). Typical values 
of the watermark length N and the finmes per watermark san^le K are 1 (K2A and S, 
respecth^ely. According]^, the first bufEer 31 contains estimated watermadc sanqples v(l), 
v(6), v(l 1), .., the second buffer 32 contains v(2), v(7), v(12), .., etc. This implies that the 
granularity of watermark detection is approximately 3 minutes and 25 seconds for PAL 
video. 

The watermark is detected by determining the similarity of the contents of 
each buffer with the a reference watermark w(n) being looked for. Each watermark can for 
instance identify one theatre. A well-known example of similarity is cross-correlation, but 
oiher m^isures are possible. The contents of each buffer is cross-correlated with the reference 
watermark in respective correlators 33, 34, ... The correlation is preferably performed using 
Symmetrical Phase Only Matched Filtering (SPOMF). For a description of S 
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reference is made to Intetnatiosal Patent plication WO 99/45706. In said document, the 
conelation is peifonned in the two-dimensional spatial domain. Blocks of NxN image pixels 
aie cocrelated with an NxN reference watermattc The result of the SPOMF operation is an 
NxN pattern of conelation values e^iibiting one or more peaks if a watermark has been 
erobedded. 

The K correlators 33, 34, ^ operate in the one-dimensional time domairu The 
ou^ut of each correlator is a series of N correlation values which is stored in a corresponding 
one of K buffers 35, 36, ... A peak detector 37 searches the highest correlation value in the K 
buffers, and ^lies said peak value to a threshold circuit 38. If the peak vabxe of at least one 
of the buffers is larger then a given threshold value, it is decided that die watemiark is 
present Otherwise the content will be classified as not watermarked. 

A payload can be encoded in the sdgnal by embedding shifted versions of the 
watennaik w(n) in a manner similar to the one disclosed in International Patent Application 
WO-A-99/4S70S. It should furdier be noted that, although parallel correlators are shown in 
Fig. 3, it may be advantageous to carry out the respective operations in a time-sequential 
manner. 

In order to describe the invention in even more details, a mathematical 
analysis of the prior art watermaddng scheme, an analysis of the publicly available de- 
fhckering tool, and the operation of the watermarking scheme in accordance with the 
invention will now be givea 

The watermark w is a periodic pseudo-random sequence containing only * 1 * 
and *-l • sanq)le values witii period M. A watermark sample w(n) is embedded in K 
consecutive frames k, k+1,.., k+K-1 . By embedding one watennaik sample in K consecutive 
frames, the frequency of the flickering due to the embeddmg is decreased. A * 1 ' is enibedded 
in an image by increasing the luminance value of each pixel wifli a value CF(n,k) . A * - 1 ' is 
embedded by decreasing llie lummance value of each pbcel with C^(n^. Herem, n is the 
spatial coordinate of a pixel within fiame k. Ivlbre mathematically we have: 

F (nk) = l^^^^^^^^^^ ifwLkA'J = l 
lF(n,k)-CF(n,k) ifwLk/rJ = -l 

where F is the firame to be embedded and Fw die embedded frame. The change Cp is chosen 
such that the watermark is not visible, and depends therefore on F. In [2], a texture detector 
and a motion detector are used to determine Cf. As a result, the mean luminance values of die 
watermarked video ^ 
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with N the ninriber of pixels per ftame/i?wn 
mean luminance values 

f^(k) = f(k)+CF(k)w(Lk/Tj) (1) 

Herein, cf is the local depth of the watermark w, which is directly related to Cf: 

Fig. 4 shows at (a) a graph of flie mean luminance values of an original 
sequence and at (b) a graph of the mean luminance values of an embedded sequence to 
visualize the watermark embedding concept 

Due to the watermark embedding, the mean luminance values will decease or 
increase witili respect to the original mean luminance values in time. See Eq. (1). In practice, 
the mean luminance values of a movie exhibits a low-pass nature. Therefore, the.detector 
esthnates titese luminance values of the original unwatermarked movie by low 
the mean luminance values of the watennarked movie f^. The detector estimates the 
watermark v by subtracting these low-pass filtered means from the mean luminance values of 
the watermarked movie fw, followed by a sign operation. More mathematically, 

v(k) = sign{f^(k)-(fw®g)(k)} (2) 
where 0 denotes a (one-dimensional) convolution, and g is a low-pass filter. Since a 
watemiark ssmple is embedded in T consecutive fiumes, this operation yields T esthnates 
wi of the watermark w: 

Wi(k)=:va+kK),0<l<K (3) 
Each of these K estimated watermarks wj is correlated with die original 
watermark w.Ifthe absolute correlation value di=|(w,wi>| is larger flian a threshold value, 

it is decided that the video sequence is watermarked. 

A movie is projected progressively at a firame rate of 24 frames per second 
(1|)s), howevCT a standard video camera records at 25 ^s (PAL) or 29.97 fys (NTSC) 
intedaced. Due to this interlacing, the luminance will not be the same througjiout recording 
of one ficame, as the shutter may just be cypening or justbe closing. Since the video camera 
and the projector are not synchronized, this problem is difficult to be solved fi>r a camera 
man. In addition, since the fiame rates of the projector andtiie video camera are not matched. 
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a similar problem reveals; at some points in time a fiame is recorded when the shutter is half- 
open or even fully shut The result of these mismatches is a flickering m die recorded movie. 

On Ihe Intemet, a 'de-flickei' phig-in for Virtualdub can be fimnd. This phig-m removes &e 
flickering in four stqps: 

1. In a firstpass, it calculates the mean luminance values f^ of the movie; 

2. Subsequently, it filters these means f^ with a low-pass filter h (de&ult is a smq>le 
averaging filter of length 12); 

3. Then it calculates fectors P(k:) between flie original means f^ (k) and the filtered means 
for each fiame: 

fw(k) 

4. In a second pass, the hmimance value ofeach pixel in fiame k is mult^lied with the 
OQrresfpondmg fector p(k) rounded to the nearest integer, and clipped if it exceeds the 
maximum luminance value 255. 

Note diat p(k) is non-negative for all k, since f ^ (k) >0 and h is a low-pass 
filter. If we ignore the rounding and the clipping for the moment, the result of Ihrae 
multq)lications m the last step is that the means of the new constructed video %f ^ig^ 
resembles flie low-pass filtered' means'of f^ : " 

fw.deflic = P(k)f (k) = (fw ® h)(k) 

Perceptually this means that the new constructed video exhibits less flickermg, 
sfaice flickering can be seen as a hig^ fi«quency component m the mean luminance values of 
the frames, which are now filtered out 

Unfortunalety, the watermaddng scheme is actually a flickering, altiiough 
imperceptible. As a dhect consequence, tins 'de-flickef plug-in removes the watermark. The 
watermaflc embeddmg scheme must thus be modified such that it is robust to de-flickering. 
This is all the more tue since fliis 'de-flicker* tool is widely used to undo the pirate copies 
fixnn the flickering. 

To this end, each fiame is divided into two parts (e.g. lefl/right or top^ottom), 
and the watermark sample is embedded in these parts in an opposite way. Toembeda 
watermark san^le, the mean hmiinance one part is increased and the mean lummance of the 
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other part is decreased. So the mean luminance values :^ of the watermarked movie now 
consists of two parts 

fw(k) = fw4(k)+fw;z(k) 
with(c£Eq.(l)) 
5 fw4(k) = flO^)+CF,l(k)w(Lk/Tj) and 

fw^Ck) = f2(k)-CF^(k)w(Lk/Tj) (4) 
After capturing with a camera, Ihe "de-fUcket* tool removes the flickering by 
low-pass filtering Ihe mean luminance values f^ : 

fw,deffic(k) = P(k)f^(k)|f^^(k)+fw;2(k)J 

1 0 The detection of the watermadc for the modified watermarking scheme is 

similar to the detection method described above. First tiie detector estimates the luminance 
values of the original unwatermarked movie for both parts by low-pass filtering tiie mean . 
luminance values of both parts. Then it subtracts the result of both operations fixxm the 
luminance values of tiie corresfponding parts. Finally, it makes an estunate of the watermark 

IS V by subtraction followed by a sign operation (cf. Eq. (2)): 

v(k) = siga{(fw4^defic 00 - (f w4.deflic ® g)(k) - (fw,2,defic 0^) ^ (Xwa 

®g)(k))} (5) 

The K estimations w^ (see Eq. (3)) are obtained in Ihe sinodlar way and 
correlated with the watermadc w. 

The embedded watermark survives the de-flicker operation because the de- 
20 flicker tool multiplies all the pixels of an image by tiie same £actor P(k), thereby leaving tiie 
luminance differences between the two image areas substantially intact The efifect of tiie 
invention can also be explained more mathematically. It is assumed that tiie original 
unwatermarked movie exhibits a low-pass nature. After capturing of the movie with a 

camera, the mean luminance values of tiie watermarked parts f^^ and f^^ exhibit a 

25 flickering 

f^4(k) = 7(k)f^^l(k)aiid 
fw^(k)=7(k)fw;i(k) 

Herein, Y(k)>0 corresponds to the change in tiie mean luminance value (tiie flickering) of 
frame k. The ^de-flickef plug-in removes this flickering by low-pass filtering tiie mean 
30 luminance values 

fw.deffic(k) = P(k)fw(k) = P(k)7(k)^^^^ 
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Fiom this expression it follows fhat 

Since in practice '^p^ (k) - cp^(k)jw([k/Tj) is relatively small conipared to we can 
iqppioximate p(k:)Y(k) by 1. By using ibis approximation, we see fhat 

5 fw4,demc(k) = P(k)Y(k)fwa(k)«fw40^^ 

For the other pai^ we obtain a similar result 

fwAdeffic (k) « f2(k)-CF;j(k)w(Lk/Tj) 

Using these results, we finally obtain the following expression for v (see Eq. (5)) 

v(k)«sigaifi(k)-f2(k)+[cF,i(k)+CF,2(kM 
= signtcF4(k)+CF^(k)]w(Lk/Tj)}=sign{w(L^ 

10 whereitisassumedthatfhelow-passfiltergcompletely filters out the watei^^ 

<^F4Ck) + cp^Ck) does not influence the sign of the e^nression, since it is non-negative for all 

k. From lliis expression, it can be seen that the watermark indeed survives the ^de-flickef 

operation after the modification. 

Methods and arrangements for embedding and detecting a watermark in a 
IS cinema movie are disclosed such that the watermark can be detected in a copy made by a 

handheld video camera. The watermaik embedder divides each image frame into two areas. 

A watermark bit '+1 ' is embedded in a frame by increasing the luminance of the first part and 

decreasing the luminance of Ifae second part A watermark bit '-1' is embedded by 

decreasing the luminance of tiie first part and increasing the luminance of the second part 
20 With file invration is achieved that tiie embedded watermark survives 'de-flicker' operations, 

that are ofien used to remove flicker caused by the different frame rates of cmema projection 

equipment and consumer camcorders. 



PHNL030093EPP 

11 20.01.2003 

Claims 



1. A method of embedding a watenn^ comprismg fhe 
steps of: 

- representiboig said watemiark by a sequence of watemiark samples each having a fbrst or 
second value; 

- dividing an image of said motion image signal into at least a jBrst and a second image 
area; 

- detemnning a global property of the fbrst and second image area; 

- modifying said image to increase Ibe global property of its first area and decrease the 
global property of its second area for embedding the first value of a watermark sample 
into saidinoage, and to decrease the global property of its first area and increase the 
0ohal property of its second area fixr embedding the second value of said watermadc 
sample into said image. 

2. A method as claimed in claim 1, wherein said global property is the mean 
luminance value of the respective image area. 

3. A method as claimed in claim 1, wherein said modifying step con:5)rises 
modifying series of consecutive images in accordance with fhe sanoe watermaric san^le. 

4; A method as claimed in claim 1, wherein said first and second image area are 

the upper and lower half of an image. 

5. A method as claimed in claim 1, wherein said first and second image area are 
the left andrig^t half of an image. 

6. An arrangemrat for embedding a watermadc in a motion image signal, the 
arrangement comprising: 
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- means for representing said watCTnatk by a sequence of watennaik sanqplra each having 
a fiist or second value; 

- means for dividing an image of said motion image signal into at least a first and a second 
image area; 

S - means for determining a global property of the first and second image area; 

- ima^ modifying means being airanged to increase Hie global property of the first image 
area and decrease the global property of the second image area in response to the first 
value of a watermaric sanople to be embedded into said image, and to decrease the global 
property of the first image area and increase the global property of the image second area 

10 in response to embedding the second value of a watermark sample to be embedded into 
said image. 

7. Anaielhodof detecting a watermark in a waterinaiked motioning 
signal, comprising the steps of: 

IS - dividing each image of said host signal into at least a first and a second image area; 

- determining a global property of the first and second image area; 

- computing, for each of a series of images, the difiference between the global property of 
the first and second image area; 

- correlating, for said series of images, the respective diff^ences with the watermark to be 
20 detected 

£ 

8. A method as claimed in claim 7, wherein said global property is flie mean 
luminance value of the respective image area. 

25 9. A me&od as claimed in claim 7, further mcluding the step of sid^^ 

the series of global properties a low-pass filtered version tfaereoi^ and applying the correlating 
step to the subtracted signal. 

10. A metibod as claimed in claim 9, Anther including the step of determining the 

30 sign of said subtracted signal, and ^plying the correlating step to said sign. 



PHNL030093EPP 



13 20,01.2003 

Abstract 



Methods and anangements for embedding and detecting a watennailc in a 
cinema movie such that the watennadc can be detected in a copy made by a handheld video 
camera. The watennadc embedder divides each image frame into two areas. A watemiadcbit 
*-f 1 ' is embedded in a frame by increasing Ihe luminance of the first part and decreasing the 
luminance of the second part A watMmatk bit -1' is embedded by decreasing Ihe luminance 
of the first part and increasing the luminance of the second part With the invention is 
achieved that the embedded watermark survives 'de-flicker' operations, that are often used to 
remove flicker caused by the dififerent fiame rates of cinema projection equipment and 
consumer camcorders. 
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